Location, location, location: Challenges of Outsourced Usability Evaluation

BACKGROUND This position paper presents some of the challenges experienced in relation to an outsourced usability evaluation of commercial collaboration product, which we would like to raise in the Improving the Interplay between Usability Evaluation and User Interface Design workshop. The paper describes the context of the outsourced evaluation, three challenges of location, how the evaluation was carried out and reported. Finally, we outline some of the lessons learned. The product is being developed within a multi-national software product company based in the United States. Typical of this type of company, the product company has a multitude of existing and new products under development in various programs under aggressive time and resource constraints. The company has a strong commitment to being focused on the needs of customers in relation to their products and services. As such, the company has strong human computer interaction (HCI) skills supporting the development of user interfaces that are easy to use. However, the number of these resources is limited in relation to the number of projects and amount of HCI work required. As with many companies throughout the world, this product company is investigating an outsourcing model to support HCI requirements and in particular usability evaluation. INTRODUCTION A commercial company is developing a new product, which is intended to support collaborative work amongst non-technical commercial workers. For this product to succeed, non-technical users must be able to use the tool easily. A significant component of the ease of use of the product is the users’ ability to create a clear and coherent mental model of the system. In order to evaluate the current design of the product, it was decided to conduct a usability evaluation of the current design. The overall objective for the usability test was to determine whether the product supports a coherent and consistent mental model for a user collaboratively sharing files with others to achieve a goal. The secondary goal of the evaluation was to determine whether the interface screen design and flow supports the individual tasks of creating and sharing through the product with another person and accepting an invitation to share. The company has offices in Australia that, aside from dayto-day business are involved in HCI based research in collaboration with the Universities of Melbourne and Aalborg. This program has been running for over four years encompassing collaboration on developing research techniques, industry projects, teaching and sponsorship of a state of the art usability laboratory in The University of Melbourne, Department of Information Systems. The university has strong skills and resources in usability evaluation and is very active in research and teaching of evaluation techniques. The evaluation allowed us to study some of the challenges of outsourcing usability in a large industrial software development project. In the following sections, we first briefly introduce the context of the product usability evaluation. Secondly, we outline some of the challenges encountered in planning and conducting the evaluation, which we would like to address in the workshop. Hereafter, we briefly describe how the evaluation was carried out and the mechanisms employed for reporting them. Finally, we outline some of the lessons learned. Given this collaborative relationship, the company decided to pilot a set of usability tests on one of their developing products using the Company – Melbourne – Aalborg relationship. Following discussions with senior company product managers, the product was selected as a suitable candidate based on it being at an appropriate state of development, requiring HCI support and being an open source development which circumvented non-disclosure requirements. From the company perspective, the objective of the testing was to determine whether cost effective useful findings